New! Sign up for our free email newsletter.
Science News
from research organizations

Score, then rank: Researchers propose an integrated approach to grant review assessments

Date:
August 1, 2023
Source:
American Institute of Biological Sciences
Summary:
The public funding of science is responsible for many of the biomedical and other scientific breakthroughs on which our lives depend. However, the process through which funding decisions are made, the peer review of grant proposals, has been historically understudied, and current approaches can lead to undesirable outcomes.
Share:
FULL STORY

The public funding of science is responsible for many of the biomedical and other scientific breakthroughs on which our lives depend. However, the process through which funding decisions are made, the peer review of grant proposals, has been historically understudied, and current approaches can lead to undesirable outcomes. Writing in Research Integrity and Peer Review, Stephen A. Gallo, then affiliated with the American Institute of Biological Sciences, and Michael Pearce, Carole J. Lee, and Elena A. Erosheva from the University of Washington highlight problems related to proposal rating in grant review and propose a protocol to ameliorate them.

In many review programs, say the authors, proposals are funded primarily on the basis of reviewer ratings and summary statistics derived from them that represent reviewers' assessment of proposal quality. Relying solely on ratings, however, can pose problems for establishing funding priorities, "especially when distinguishing between similarly rated proposals, especially for projects of high quality, where only a few minor weaknesses can make the difference between a funded project and a rejection." Furthermore, say Gallo and colleagues, the use of ratings rubrics "often forces reviewers to penalize a project based on a series of identified but minor weaknesses, despite the fact that they may find the application overall to be potentially significant."

To address these concerns, the authors suggest adding rankings to the traditional rating approach and describe an innovative methodology called the Mallows-Binomial, which can then produce integrated proposal scores, as well as an induced preference ordering. In so doing, the approach allows funders to consider reviewers' scores alongside their opinions about how well the proposals stack up against one another. Pearce states that the inclusion of ranking will "more accurately distill reviewer opinion into a useful output to make an informed funding decision."

For research funders, the authors suggest clear benefits of the Mallows-Binomial protocol in comparison with ratings-only approaches. These include "providing a ranked priority list with confidence metrics, a higher degree of discrimination between similarly rated proposals, and a robustness to outliers and reviewer inconsistencies," features that will be important in distilling reviewer opinion "into a useful output to make the best, most informed decision" for funding crucial research.


Story Source:

Materials provided by American Institute of Biological Sciences. Note: Content may be edited for style and length.


Journal Reference:

  1. Stephen A. Gallo, Michael Pearce, Carole J. Lee, Elena A. Erosheva. A new approach to grant review assessments: score, then rank. Research Integrity and Peer Review, 2023; 8 (1) DOI: 10.1186/s41073-023-00131-7

Cite This Page:

American Institute of Biological Sciences. "Score, then rank: Researchers propose an integrated approach to grant review assessments." ScienceDaily. ScienceDaily, 1 August 2023. <www.sciencedaily.com/releases/2023/08/230801152846.htm>.
American Institute of Biological Sciences. (2023, August 1). Score, then rank: Researchers propose an integrated approach to grant review assessments. ScienceDaily. Retrieved October 31, 2024 from www.sciencedaily.com/releases/2023/08/230801152846.htm
American Institute of Biological Sciences. "Score, then rank: Researchers propose an integrated approach to grant review assessments." ScienceDaily. www.sciencedaily.com/releases/2023/08/230801152846.htm (accessed October 31, 2024).

Explore More

from ScienceDaily

RELATED STORIES